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ABSTRACT 

Over  the  years  a  great  deal  of  multi-sensor  fusion  research  has  been  conducted.  While  the  potential 
military  value  of  such  research  is  considered  high,  there  have  been  few  fielded  examples  in  the  tactical 
arena.  The  objective  of  the  Multi-Spectral  Sensor  Surveillance  System  (M4S)  program  is  to  transition 
high-payoff,  low  risk  multi-sensor  fusion  technology  from  the  research  and  development  phase  into 
practical  user-oriented  systems  applications. 

The  primary  application  focus  of  M4S  is  toward  a  robust  and  highly  integrated  autonomous  littoral 
surveillance  capability  for  either  manned  or  unmanned  surveillance  aircraft.  Sensor  data  from  various 
organic,  i.e.  on-board,  sensors  as  well  as  off-board  sources  is  used  to  provide  superior  battlespace 
awareness.  Specific  operational  payoffs  from  M4S  include  reduced  manning  requirements,  improved 
detection,  classification,  and  geo-location  accuracy,  and  reduced  prosecution  timelines  against  time- 
critical  targets. 

This  objective  and  focus  has  led  to  the  selection  of  a  suite  of  notional  sensors  and  a  data  fusion 
architecture  consistent  with  the  limited  space  and  computational  capabilities  of  airborne  platforms.  The 
four  non-developmental  organic  sensors  chosen  for  M4S  include  the  PLAID  ESM  system,  AN/APS-137B 
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(V)  5  synthetic  aperture  radar,  AN/AAQ-49  passive  mid-wave  FLIR,  and  a  coherent  C02  LIDAR 
vibration  sensor.  The  distributed  data  fusion  architecture,  which  includes  closed  loop  control  of  the  above 
on-board  sensors,  is  implemented  using  Pentium  class  microprocessors  and  the  Windows  NT  operating 
system.  All  internal  communications  between  system  components  is  provided  by  a  Multipoint  Messaging 
System  (MMS),  which  provides  information  flow  control  via  an  ask-answer  messaging  protocol.  Use  of 
MMS  has  aided  system  integration  and  provides  a  plug-and-play  capability  for  new  or  additional  sensors. 

To  facilitate  initial  system  development  and  performance  analysis,  a  System  Integration  Laboratory  (SIL) 
has  been  established.  The  SIL  provides  a  very  cost  effective  development,  integration  and  test 
environment  that  includes  the  capability  to  extract  internal  system  component  data  during  real-time 
operations.  This  land-based  test  bed  is  capable  of  providing  simulated  sensor  operation  in  representative 
operational  scenarios.  Scenario  simulation  capabilities  also  include  various  target,  background  and 
propagation  models,  as  well  as  noise  and  other  probabilistic  effects  to  permit  realistic  system  evaluation. 
A  test  controller  in  the  SIL  manages  experiments  and  accumulates  data  for  off-line  Monte  Carlo  analysis 
of  system  and  data  fusion  performance  metrics.  The  SIL  has  been  constructed  so  that  it  can  be  integrated 
with  real  sensors  for  actual  flight  testing.  Proof  of  concept  demonstrations  were  successfully  conducted 
in  the  SIL  in  October  1 997  using  various  over-water  and  over-land  scenarios  in  the  Persian  Gulf  area. 


1.0  INTRODUCTION 

Over  the  years  a  great  deal  of  multi-sensor  fusion  research  has  been  conducted.  While  the  potential 
military  value  of  such  research  is  considered  high,  there  have  been  few  fielded  examples  in  the  tactical 
arena.  The  objective  of  the  Multi-Spectral  Sensor  Surveillance  System  (M4S)  program  is  to  demonstrate 
the  operational  utility  of  such  of  such  a  system,  quantifying  improvements  to  the  maximum  extent 
possible.  In  this  context,  M4S  is  addressing  not  only  the  information  fusion  from  the  outputs  of  a  suite  of 
dissimilar  sensors,  but  also  a  closely  integrated  interaction  of  these  sensors,  as  typified  by  human  sensor 
processes.  As  an  example,  it’s  one  sensor  telling  another  in  which  general  geographic  area  to  search  for  a 
target,  as  well  as  give  it  some  idea  of  what  it’s  looking  for.  Or,  as  another  example,  M4S  realizing  it 
needs  a  particular  piece  of  critical  information  to  make  a  positive  ID  on  a  target.  It’s  our  belief  that  there 
are  some  significant  gains  in  ISR  capability  that  are  available  even  with  our  existing  sensor  inventory...  if 
we  can  get  these  sensors  to  interact  more  synergistically. 

To  implement  this  concept  in  an  airborne  surveillance  context,  M4S  has  selected  a  notional  airborne 
surveillance  platform  (see  figure  1 )  which  could  be  either  a  manned  or  unmanned  aircraft,  and  identified 
four  off-the-shelf  sensors,  including  an  ESM,  SAR,  FLIR,  and  LADAR,  which  will  be  later  described  in 
further  detail.  While  M4S  could  also  take  advantage  of  off-board  sensor  information  as  well,  the  initial 
efforts  have  focused  on  the  use  of  these  organic  sensors  in  littoral  environments,  as  might  be  the  case  with 
an  organic  surveillance  asset  under  the  control  of  a  joint  theatre  commander. 


Figure  1  Multi-Spectral  Sensor  Surveillance  System  (M4S)  Concept 


2.0  OPERATIONAL  IMPACT 

Operational  benefits  that  can  be  achieved  through  implementation  of  this  data  fusion  technology  in  an 
operational  context  include:  Enhanced  real-time  awareness  of  friendly,  neutral,  and  adversary  forces; 
improved  detection,  classification,  identification  and  battle  damage  assessment;  Reduced  operator 
workload  by  increasing  automation  of  information  processing  and  sensor/flight  control;  improved 
prosecution  of  time  critical  targets,  i.e.  tighter  sensor  to  shooter  connectivity;  and  reduced  off-board 
communication  bandwidth  requirements  through  increased  on-board  processing. 


3.0  TECHNICAL  APPROACH 

The  M4S  technical  approach  is  to  use  data  fusion  technology  to  exploit  the  synergy  of  our  four-sensor 
suite.  That  synergism  comes  from  a  number  of  factors,  including  the  broad  spectral  coverage,  as  well  as 
measurement  of  different  target  phonologies.  Indeed,  M4S  covers  not  only  the  EM  and  EO  portions  of 
the  spectrum,  but  also  the  acoustic  end  via  the  vibrometry  information  obtained  from  the  LADAR.  That 
allows  not  only  the  ability  to  take  full  advantage  of  any  emissions  or  characteristics  of  a  particular  target, 
but  also  allows  mitigation  of  adverse  environmental  or  propagation  effects.  Additionally,  the  M4S 
concept  can  be  extended  to  include  multiple  distributed  platforms  and  sensor  systems. 


4.0  SELECTED  SENSORS 


Figure  2  depicts  the  specific  sensors  chosen  for  the  M4S  sensor  suite.  They  include,  starting  in  the  upper 
left  and  working  clockwise:  the  multimode  AN/APS-137  Synthetic  Aperture  Radar;  a  2-1 8  GHz  Passive 
Location  and  Identification  ESM  system  currently  being  procured  by  the  Air  Force,  and  which  provides 
passive  ranging  and  SEl  capability;  an  AN/AAQ-49  mid-wave  FLIR  used  by  the  LAMPS  helicopter;  and 
a  coherent  10.6  micron  C02  LADAR  that  provides  target  vibration  information. 


Figure  2  Selected  M4S  Sensor  Suite 

5.0  M4S  ARCHITECTURE 

Figure  3  provides  a  graphical  depiction  of  the  overall  M4S  system  architecture.  At  the  far  left  in  the 
graphic  are  the  individual  sensors,  which  each  feed  their  own  Sensor  Processor.  The  Sensor  Processor 
provides  both  a  software  and  hardware  interface  between  the  individual  sensors  and  the  M4S  system, 
allowing  the  transfer  of  sensor  data  into  M4S,  as  well  as  sensor  operation,  i.e.  pointing  and  mode  control, 
by  the  resource  manager.  Additionally,  it  provides  a  large  measure  of  the  system’s  plug  and  play 
capability,  should  additional  or  alternative  sensors  be  used  with  M4S. 


M4S  ARCHITECTURE 


Figure  3  M4S  Architecture 

The  outputs  of  the  Sensor  Processors  each  feed  individual  Sensor  Trackers.  Each  Sensor  Tracker  takes 
individual  sensor  detections  and  observed  target  characteristics,  and  attempts  to  establish  mature  tracklets, 
or  short  track  segments,  which  include  not  only  target  positional  information,  but  target  characteristics 
such  as  target  length  or  vibration  spectrum. 

The  Sensor  Track  Fuser,  which  is  based  upon  the  Advanced  Tactical  Workstation,  is  where  all  the 
dissimilar  sensor  source  information  is  first  brought  together.  There  the  fuser  attempts  to  correlate,  both 
spatially  and  temporally,  tracklet  information  originating  from  any  single  type  sensor,  or  cross  correlate 
information  from  dissimilar  type  sensors.  The  output  from  this  process  is  a  track  database  that  provides  a 
complete  description  of  the  time  history,  target  characteristics,  and  possible  identification  of  all 
maintained  tracks.  The  fuser  is  a  statistical  tracker  that  makes  soft  decisions,  and  as  such  is  able  to  go 
back  and  revise  merge/demerge  decisions,  or  handle  out-of-order  data. 

Providing  a  situation  understanding  capability  in  M4S  is  a  new  effort  just  getting  started.  Here  again 
M4S  uses  a  database,  this  time  of  known  enemy  scenarios  and  behaviors,  to  infer  a  higher  level  of 
understanding  of  the  threat’s  intent  and  possible  future  actions. 

The  last  block  in  the  M4S  architecture  is  the  Resource  Manager,  which  provides  pointing  and  mode 
control  of  the  individual  sensors,  as  well  as  flight  path  waypoint  updates.  It  should  be  noted  that  all 
communications  within  M4S  take  place  via  a  Multipoint  Messaging  System  -  an  ask/answer  messaging 
format  that  throttles  the  flow  of  information  within  the  M4S  system. 


6.0  Simulation 


Since  M4S  is  a  development  program,  it  was  realized  early  that  a  virtual  or  synthetic  environment  was 
needed  to  support  M4S.  Such  an  environment  could  support  software  development,  integration  and 
laboratory  testing  before  heading  out  into  the  much  more  complex  real  world.  It  would  also  permit  the 
conduct  of  highly  repeatable  Intelligence,  Surveillance,  and  Reconnaissance  (ISR)  missions  in  any  desired 
area  of  the  world  in  support  of  Monte  Carlo  evaluation  of  Measures  of  Effectiveness  (MOE)  or  Measures 
of  Performance  (MOP).  The  capabilities  listed  in  figure  4  identify  some  of  the  present  and  future  planned 
M4S  sensor  modeling  capabilities. 


SIMULATION  -  SENSOR  MODELS 
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Figure  4  Sensor  Modeling  Capabilities 


In  addition  to  modeling  the  sensors  themselves,  M4S  has  also  developed  all  the  necessary  components  to 
conduct  and  evaluate,  again  through  Monte  Carlo  techniques,  several  typical  ISR  missions.  These 
modeled  components  include  the  earth,  various  target  arrays,  the  airborne  surveillance  platform,  and 
atmospheric  phenomenon  such  as  weather  effects.  These  components  are  listed  in  figure  5. 


SIMULATION  -  SCENARIO  MODELS 
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Figure  5  Scenario  Modeling  Capabilities 

7.0  System  Integration  Lab  (SIL) 

Figure  6  is  a  photograph  of  the  M4S  System  Integration  Lab,  or  SIL,  where  software  development  and 
test  takes  place,  and  virtual  ISR  missions  are  conducted.  The  M4S  architecture  is  implemented  using  a 
Windows  NT  operating  system  hosted  on  Pentium  class  machines,  networked  together  over  an  Ethernet. 
Also  noted  in  the  graphic  there  is  a  test  controller  that  conducts  the  Monte  Carlo  simulations,  and  records 
appropriate  data  for  post  mission  MOE/MOP  analysis. 


SYSTEM  INTEGRATION  LAB  (SIL) 
IS  UP  AND  RUNNING 


Figure  6  System  Integration  Lab  (SIL) 


Three  ISR  scenarios  have  been  implemented  in  the  SIL  to  demonstrate  various  M4S  functional 
capabilities.  These  scenarios  include  1)  An  ocean  surveillance  scenario  which  demonstrates  the  handoff 
from  off-board  to  onboard  sensors,  the  ability  to  track  multiple  targets,  and  target  identification;  2)  An 
armored  column  scenario  which  demonstrates  the  integration  of  intelligence  information,  and  dynamic 
flight  path  control;  and  3)  A  site  assessment  scenario  which  demonstrates  the  ability  to  separate  and 
identify  closely  spaced  targets,  and  operate  in  the  presence  of  a  surface  to  air  missile  threat.  All  of  these 
scenarios  take  place  in  the  Persian  Gulf  near  the  Straight  of  Hormuz. 


8.0  M4S  Program  Schedule 


Figure  7  provides  the  overall  M4S  program  schedule.  As  can  be  noted,  M4S  is  still  in  laboratory 
development,  but  is  aiming  for  an  actual  flight  test  of  the  system  in  a  manned  aircraft,  such  as  a  P-3,  in  the 
FYOl  timeframe. 


M4S  SCHEDULE 


Figure  7  M4S  Program  Schedule 

9.0  Summary 

In  summary,  M4S  is  a  unique  program  that  can  provide  significant  improvements  in  ISR  capability, 
baselined  using  existing  sensors,  while  at  the  same  time  reducing  current  operator  workload.  A  critical 
next  step  for  the  M4S  program  is  to  start  receiving  feedback  from  the  operational  user  community, 
especially  regarding  their  thoughts  on  measures  of  performance  and  measures  of  effectiveness. 
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